Repository navigation
Write Parquet NUMERIC with precision up to 76 and name the array column in overflow messages - #90
Conversation
huiliang-liu
changed the title
eefcc59 to
9dbd712
Compare
huiliang-liu
changed the title
|
One small thing: PG allows |
Good catch, thanks. Fixed in parseDecimalStringWithBigDecimal: NaN is now stored as NULL under ignore (same as the ≤ 38 path), and error / round raise a clear "is not a number" error instead of a raw NumberFormatException. Added unit tests for all three options. I left the ≤ 38 path unchanged here to keep this PR's "precision ≤ 38 is unchanged" scope, and will file a follow-up for its misleading messages. |
PXF rejected any NUMERIC column with a precision above 38 when writing Parquet, because values were converted through HiveDecimal. Parquet's DECIMAL has no such limit, and Arrow-based readers such as PyArrow read precision up to 76 as decimal256. Columns with a precision from 39 to 76 are now parsed with BigDecimal and written as a FIXED_LEN_BYTE_ARRAY wide enough for the precision. The pxf.parquet.write.decimal.overflow option keeps its error, round and ignore behavior for these columns. Precision up to 38 still goes through HiveDecimal, and NUMERIC without a precision still maps to DECIMAL(38,18), so existing files and readers limited to 38 digits are unaffected.
When a NUMERIC[] element did not fit the Parquet decimal type, ParquetResolver looked up the column name with the element's index inside the repeated group (always 0), so the overflow error, or the warning with pxf.parquet.write.decimal.overflow=ignore, named the table's first column instead of the array column. Pass the column name down from fillGroup instead of deriving it from the primitive's index. Add tests for how the scale of a NUMERIC[] column is chosen on write, from a field report where values with more than 18 decimal places were rounded. The scale comes from the writable table column: NUMERIC[] falls back to DECIMAL(38,18), NUMERIC(38,20)[] and NUMERIC(38,37)[] keep their scale, NUMERIC(76,61)[] and TEXT[] keep every digit, and NUMERIC(77,61)[] is rejected. Unit tests cover the resolver; the automation test covers the same cases end to end.
PG allows NaN in a constrained NUMERIC(p,s) column, but a Parquet DECIMAL cannot hold it. For precision 39-76, parseDecimalStringWithBigDecimal passed NaN to new BigDecimal() and failed with a raw NumberFormatException under every overflow option. NaN is now stored as NULL under 'ignore', matching the precision <= 38 path, and 'error' / 'round' reject it with a message saying the value is not a number.
The FDW sent type modifiers only for scalar columns, so PXF never saw the precision and scale of a NUMERIC(p,s)[] column, or the length or precision of the other array types the external table already covers. A NUMERIC(76,61)[] written through a foreign table was stored as DECIMAL(38,18), rounding its values to 18 decimal places. Handle the array OIDs the same way external-table/src/pxfheaders.c does, with the same locally defined OIDs, since the WHPG 6 catalog headers do not define them.
060b109 to
52f4414
Compare
JIRA: PTT-1957
Summary
Writing a
NUMERICvalue with more than 38 digits to Parquet loses digits or fails. In a field report,numeric[]values with more than 18 decimal places came back rounded (0.1234567890123456789→0.123456789012345679, and a value with 61 decimal places →0), and declaring a precision above 38 was rejected. The customer needs these values asdecimal256on the Parquet side.This PR has two commits.
1. Write Parquet NUMERIC columns with precision up to 76
NUMERIC(p,s)withp > 38because values went throughHiveDecimal. Parquet'sDECIMALhas no such limit, and Arrow readers (PyArrow) read precision up to 76 asdecimal256.ParquetFileAccessor.MAX_DECIMAL_PRECISION). For precision 39–76, values are parsed withBigDecimal(DecimalUtilities.parseDecimalStringWithBigDecimal) and written as aFIXED_LEN_BYTE_ARRAYsized for the precision (32 bytes at 76).pxf.parquet.write.decimal.overflowkeeps itserror/round(half-up) /ignore(NULL) behavior for these columns.HiveDecimal, so those files are byte-for-byte the same.NUMERICwithout a typmod still maps toDECIMAL(38,18). Spark, Hive and Trino only read precision ≤ 38, so wider columns are opt-in by declaring the precision.BigDecimal. No change was needed there.hdfs_parquet,cfg_server) and CHANGELOG are updated.2. Name the array column in Parquet numeric overflow messages (from #89)
NUMERIC[]element that overflowed, the error (or theignore-mode warning) named the table's first column instead of the array column. The column name is now passed down fromfillGroup.NUMERIC[]column's scale is chosen on write.NUMERIC[]→DECIMAL(38,18).(38,20)and(38,37)keep their scale.(76,61)andTEXT[]keep every digit.(77,61)is rejected.NUMERIC(45,25)[]to be rejected now checks a(76,61)round trip instead.Testing
pxf-hdfsunit tests: 539 passed. New tests are inParquetWriteTestandDecimalUtilitiesTest.file:parquet:numeric(76,61)andnumeric(76,61)[]write and read back exactly, including 61-decimal-place and 15-integer-digit values, negatives and NULLs.decimal256(76, 61)with the correct values.decimal256(76, 61)file written by PyArrow reads through PXF exactly.parquetWriteNumericWithMaxPrecision(numeric_max_precision).parquetWriteNumericArrayScale(numeric_array_scale).numeric_with_large_precisionchanged from 38 to 76.numeric_max_precisionandnumeric_array_scalepass underpxf_regresslocally.numeric_with_large_precisionwas not re-run.mvn test-compilepasses. The full HDFS automation suite was not run.